Utilizing a critical edge detection neural network and a geometric model to determine camera parameters from a single digital image

ABSTRACT

The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing a critical edge detection neural network and a geometric model to determine camera parameters from a single digital image. In particular, in one or more embodiments, the disclosed systems can train and utilize a critical edge detection neural network to generate a vanishing edge map indicating vanishing lines from the digital image. The system can then utilize the vanishing edge map to more accurately and efficiently determine camera parameters by applying a geometric model to the vanishing edge map. Further, the system can generate ground truth vanishing line data from a set of training digital images for training the critical edge detection neural network.

BACKGROUND

Recent years have seen significant improvements in the field of digitalimage analysis and editing. Due to advances in algorithms and hardware,conventional digital image analysis systems are now able to analyze andedit digital images in a variety of ways. For example, conventionaldigital image analysis systems can reproject digital images to make themappear visually aligned (e.g., upright) and add or subtract objectsportrayed in digital images. Indeed, conventional digital image analysissystems can (with camera parameters in hand) size and orient a newdigital object to fit within a three-dimensional scene portrayed in thedigital image. In such three-dimensional compositing it is critical tohave a precise estimate of the camera calibration to ensure that theforeground and background elements have matching perspective distortionin the final rendered image.

Although conventional digital image analysis systems have progressed inrecent years, they still have several significant shortcomings inrelation to accuracy, flexibility, and efficiency. For example, someconventional digital image analysis systems can determine cameraparameters and modify digital images utilizing a convolutional neuralnetwork. In particular, conventional digital analysis systems can traina convolutional neural network to identify camera parameters from adigital image. Such systems, however, are not very precise and/oraccurate. Indeed, many digital image analysis systems that utilizeconvolutional neural networks determine camera parameters have a highrate of error. As a result, such systems also generate modified digitalimages that are not accurate, realistic, or visually appealing.

Some conventional digital image analysis systems utilize geometricapproaches to determine camera parameters from digital images. Forexample, such conventional systems can analyze geometric shapes in thedigital image to identify edges and determine camera parameters at thetime the digital image was captured based on the identified edges. Suchsystems, however, are not robust or flexible. Indeed, conventionalsystems that utilize geometric approaches have significant problems withaccuracy when analyzing digital images that contain misleading orconfusing shapes and/or lines (e.g., lack strong vanishing lines). Forinstance, digital images that contain a variety of round objects, curvedobjects, or lines that point in random directions can undermine theaccuracy of geometric models.

Some conventional systems can determine camera parameters by analyzingmultiple digital images of the same subject (e.g., the same scene orobject). Such systems are inefficient, as they require extensivecomputer resources and processing of multiple images to extract a set ofcamera parameters. Further, such systems offer little flexibility, asthey require digital images conforming to specific criteria, and in manycases a user is unlikely to possess the requisite images (i.e., a largenumber of digital images portraying the same subject) for the system towork properly. In addition, many systems require extensive computerprocessing resources and time to generate and utilize training data.

These along with additional problems and issues exist with regard toconventional digital image analysis systems.

BRIEF SUMMARY

Embodiments of the present disclosure provide benefits and/or solve oneor more of the foregoing or other problems in the art with systems,non-transitory computer-readable media, and computer-implemented methodsfor determining camera calibration and generating enhanced digitalimages utilizing a critical edge detection neural network and geometricmodel. In particular, the disclosed systems can accurately, efficiently,and flexibly determine camera parameters, such as focal length, pitch,roll, and yaw, based on a single digital image. For example, in one ormore embodiments, the disclosed systems utilize a deep-learning basedframework to weight edges in a digital image (e.g., to identifyvanishing lines that are relevant to digital image perspective). Thedisclosed systems can then utilize the vanishing lines in conjunctionwith a geometric model to accurately identify camera parameters andgenerate a modified digital image. To further enhance efficiency andaccuracy, the disclosed system can also generate accurate training datafrom existing digital image repositories and use this training data totrain a critical edge detection neural network. Specifically, thedisclosed systems can generate ground truth vanishing lines fromtraining digital images and utilize the ground truth vanishing lines totrain a critical edge detection neural network to identify criticaledges. Upon training, the system can utilize the critical edge detectionneural network in conjunction with a geometric model to more accuratelydetermine camera parameters and generate a modified digital image.

Additional features and advantages of one or more embodiments of thepresent disclosure are outlined in the description which follows, and inpart will be obvious from the description, or may be learned by thepractice of such example embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description provides one or more embodiments withadditional specificity and detail through the use of the accompanyingdrawings, as briefly described below.

FIG. 1 illustrates a diagram of an environment in which a cameraparameter determination system can operate in accordance with one ormore embodiments.

FIG. 2 illustrates an overview of determining camera parameters for adigital image in accordance with one or more embodiments.

FIGS. 3A-3C illustrate a process of generating ground truth vanishinglines from a set of training digital images in accordance with one ormore embodiments.

FIG. 4 illustrates a flowchart for training a critical edge detectionneural network in accordance with one or more embodiments.

FIG. 5 illustrates utilizing a critical edge detection neural network togenerate a vanishing edge map in accordance with one or moreembodiments.

FIG. 6 illustrates utilizing camera parameters of a digital image togenerate an enhanced digital image in accordance with one or moreembodiments.

FIG. 7 illustrates a block diagram of a camera parameter determinationsystem in accordance with one or more embodiments.

FIG. 8 illustrates a flowchart of a series of acts for utilizing acritical edge detection neural network to determine camera parameters inaccordance with one or more embodiments.

FIG. 9 illustrates a flowchart of a series of acts for training acritical edge detection neural network in accordance with one or moreembodiments.

FIG. 10 illustrates a block diagram of an example computing device inaccordance with one or more embodiments.

DETAILED DESCRIPTION

This disclosure describes one or more embodiments of a camera parameterdetermination system that can utilize a critical edge detection neuralnetwork in conjunction with a geometric model to identify cameraparameters from a single digital image. The camera parameterdetermination system can train a critical edge detection neural networkto identify vanishing lines in a digital image and generate an edge map(e.g., an edge map that weights vanishing lines that indicateperspective within the digital image). The camera parameterdetermination system can then utilize a geometric model to analyze theedge map generated by the critical edge detection neural network andidentify camera calibration. The camera parameter determination systemcan further improve accuracy and efficiency by generating ground truthdata for training the critical edge detection neural network.

To illustrate, the camera parameter determination system can generateground truth vanishing lines from a set of training digital images.Specifically, in one or more embodiments the camera parameterdetermination system identifies lines (e.g., edges) in a trainingdigital image and determines vanishing points by analyzing the lines.Based on distances between the vanishing points and the lines in thedigital image, the camera parameter determination system can identifyground truth vanishing lines. Then, the camera parameter determinationsystem can utilize the training digital images and the ground truthvanishing lines to train a critical edge detection neural network toidentify vanishing lines in digital images. When the critical edgedetection neural network is trained, the camera parameter determinationsystem can utilize the critical edge detection neural network togenerate a vanishing edge map that indicates vanishing lines for adigital image. In one or more embodiments, the camera parameterdetermination system utilizes the vanishing edge map (e.g., by applyinga geometric model) to more accurately and efficiently determine cameraparameters for the digital image.

As mentioned above, in some embodiments the camera parameterdetermination system identifies ground truth vanishing lines for a setof training images. More specifically, the camera parameterdetermination system can map a digital image onto a sphere and dividethe sphere into regions or “bins.” Then, the camera parameterdetermination system can identify each of the intersections of two ormore lines on the sphere. In one or more embodiments, the cameraparameter determination system utilizes a distance-based voting schemeamong the lines, intersections, and/or pixels of the digital image todetermine the major vanishing points of the image. In some embodiments,the camera parameter determination system utilizes the major vanishingpoints to determine, based on distances between the various lines in thedigital image and the major vanishing points, whether each line in thedigital image is a ground truth vanishing line.

Additionally, as discussed above, the camera parameter determinationsystem can train a critical edge detection neural network. Inparticular, in one or more embodiments the camera parameterdetermination system utilizes the ground truth vanishing lines for thetraining images to train a critical edge detection neural network todetermine vanishing lines in a supervised manner. Specifically, asdiscussed in greater detail below with regard to FIG. 4, the cameraparameter determination system can utilize a loss function inconjunction with the ground truth vanishing lines to train the criticaledge detection neural network to identify vanishing lines in a digitalimage. In this manner, the camera parameter determination system cantrain a critical edge detection neural network to identify vanishinglines based on contextual information analyzed at various levels ofabstraction within the digital image.

Further, the camera parameter determination system can utilize a trainedcritical edge detection neural network to identify vanishing lines in adigital image and generate a vanishing edge map. In one or moreembodiments, once the critical edge detection neural network is trained,the camera parameter determination system can utilize the critical edgedetection neural network to generate a vanishing edge map that comprisesa plurality of edge weights. Specifically, the vanishing edge map caninclude a weight for each pixel that indicates a probability that eachpixel corresponds to a vanishing line in the digital image.

The camera parameter determination system can utilize a vanishing edgemap to determine camera parameters for a corresponding digital image.More specifically, the camera parameter determination system can apply ageometric model to the vanishing edge map to accurately determine focallength, pitch, roll, and yaw. Indeed, because the camera parameterdetermination system can apply the geometric model to the vanishing edgemap (rather than a host of misleading or erroneous lines contained inthe digital image), the camera parameter determination system canimprove the accuracy of resulting camera parameters.

The camera parameter determination system can also utilize determinedcamera parameters to perform various functions with a digital image. Forexample, the camera parameter determination system can generate anenhanced digital image. More specifically, the camera parameterdetermination system can accurately and seamlessly add objects to athree-dimensional scene portrayed in a digital image. The cameraparameter determination system can also reproject digital images toalign vertical and horizontal lines to vanishing points. Moreover, thecamera parameter determination system can generate/estimate athree-dimensional model of a scene or object portrayed in a digitalimage based on the camera parameters and the digital image.

The camera parameter determination system provides many advantages andbenefits over conventional systems and methods. For example, the cameraparameter determination system can improve accuracy relative toconventional systems. Indeed, by applying a critical edge detectionneural network to generate accurate vanishing lines and then utilizing ageometric model to analyze the vanishing lines, the camera parameterdetermination system can accurately determine camera parameters andgenerate more accurate and realistic enhanced digital images. Thus, incontrast to conventional systems that utilize convolutional neuralnetworks that predict camera parameters, the camera parameterdetermination system can utilize a geometric model focused on analyzingaccurate vanishing lines generated by a critical edge detection neuralnetwork to produce accurate camera parameters.

In addition, the camera parameter determination system can improveflexibility relative to conventional systems. For instance, in one ormore embodiments the camera parameter determination system utilizes acritical edge detection neural network to filter out inaccurate ormisleading lines that fail to reflect image perspective. The cameraparameter determination system can then apply a geometric model to theremaining accurate vanishing lines. Thus, the camera parameterdetermination system can robustly generate accurate camera parameters ina wide variety of digital images, even digital images that includerounded shapes or random lines. The camera parameter determinationsystem can also improve flexibility by determining camera parametersbased on a single digital image (rather than requiring users to captureand provide multiple digital images of an object or scene).

Further, the camera parameter determination system can improveefficiency of implementing computing systems. In contrast toconventional systems, the camera parameter determination system candetermine camera parameters by analyzing a single digital image (ratherthan analyzing multiple digital images). In addition, the cameraparameter determination system can efficiently generate and utilizetraining data (e.g., from existing digital image repositories) fortraining the critical edge detection neural network. This cansignificantly reduce the processing power and time required to generateand utilize conventional, labeled training data.

In sum, by utilizing a critical edge detection neural network, thecamera parameter determination system can utilize contextual informationin the image to determine accurate vanishing lines (e.g. a first edge isfrom a baseboard, so the edge is likely to point to a vanishing point,or a second edge is part of a stairway banister and likely to bedistracting). Moreover, the camera parameter determination system canutilize the precision of geometric models where such models are likelyto succeed, yielding a better overall performance relative toconventional systems.

As illustrated by the foregoing discussion, the present disclosureutilizes a variety of terms to describe features and benefits of thedynamic representation management system. Additional detail is hereafterprovided regarding the meaning of these terms as used in thisdisclosure. For example, as used herein, the term “digital image” refersto any digital symbol, picture, icon, or illustration. For example, theterm “digital image” includes digital files with the following, orother, file extensions: JPG, TIFF, BMP, PNG, RAW, or PDF. The term“digital image” also includes one or more images (e.g., frames) in adigital video. Additionally, the term “digital image” refers to 3Dobjects represented in a digital format. For example, the term “digitalimage” includes digital files with the following, or other, fileextensions: OBJ, DAE, 3DS, U3D, and KMZ. Accordingly, although much ofthe description herein is phrased in terms of digital images, it will beappreciated that the disclosure can also apply to extracting propertiesfrom and/or editing digital video. Additionally, as used herein, theterm “training digital image” refers to a digital image used to train aneural network. In particular, the term “training digital image” caninclude a digital image associated with ground-truth data that can beutilized to train a neural network.

As used herein, the term “camera device” refers to any device that canbe used to capture images. In particular, the term “camera device” caninclude a device that is able to capture any kind of digital image, asdescribed above. To illustrate, a camera device can include a digital orfilm camera, a mobile phone or other mobile device, a tablet, acomputer, or any other device that can capture images.

Additionally, as used herein, the term “camera parameters” refers tocharacteristics or properties of a camera device utilized to capture adigital image. In particular, the term “camera parameters” can includecharacteristics of a camera device at the time of capture of a digitalthat impact the appearance of the digital image. To illustrate, a cameraparameter can include focal length, field of view, pitch, roll, and/oryaw.

Also, as used herein, the term “neural network” refers to a machinelearning model that can be tuned (e.g., trained) based on inputs toapproximate unknown functions. In particular, the term “neural network”can include a model of interconnected layers that communicate andanalyze attributes at varying degrees of abstraction to learn toapproximate complex functions and generate outputs based on a pluralityof inputs provided to the model. For instance, the term “neural network”includes one or more machine learning algorithms. In other words, aneural network includes an algorithm that implements deep learningtechniques, i.e., machine learning that utilizes a set of algorithms toattempt to model high-level abstractions in data. Additional detailregarding exemplary neural networks and corresponding networkarchitectures are provided below.

Further, as used herein, the term “critical edge detection neuralnetwork” refers to a neural network for identifying vanishing lines froma digital image. For example, a critical edge detection neural networkcan include a neural network (e.g., for generating a vanishing edge mapthat comprises vanishing lines from a digital image). In one or moreembodiments, the critical edge detection neural network comprises aconvolutional neural network, such as a stacked hourglass network.Additional detail regarding exemplary architectures and capabilities ofa critical edge detection neural network are discussed in greater detailbelow (e.g., with regard to FIGS. 2, 4, and 5-6).

Additionally, as used herein, the term “vanishing point” refers to aregion in an image that indicates perspective (e.g., a perspectiverelevant point or direction in which lines in an image appear toconverge). In particular, the term “vanishing point” can include apoint, vector, line, or region (e.g., an “eye point,” horizon line, ornorth pole) in a digital image where the two-dimensional perspectiveprojections of mutually parallel lines in three-dimensional space appearto converge. A vanishing point can include a horizontal vanishing point(e.g., a common direction/line/point/region along a horizon such as anAtlanta vanishing point), a vertical vanishing point (e.g., a commondirection/line/point/region for vertical lines of a digital image), orother vanishing point. A vanishing point can take a variety of forms,such as a point in two-dimensional space where lines converge. Inthree-dimensional applications (e.g., when a digital image is mapped toa three-dimensional panoramic sphere), the term vanishing point can alsoinclude a vanishing direction (e.g., a vector indicating a directionwhere perspective lines in the image point or intersect). Indeed, insome embodiments, the camera parameter determination system identifiesvanishing points that include orthogonal vanishing directions inthree-dimensional space (e.g., vanishing directions in orthogonal x, y,and z directions). Indeed, in one or more embodiments, vanishing pointscorrespond to three Manhattan directions that reflect x, y, and zdirections respectively.

Also, as used herein, the term “vanishing lines” refers to lines in adigital image that correspond to one or more vanishing points in adigital image. In particular, the term “vanishing lines” can includelines that converge at a vanishing point (e.g., at a horizon vanishingpoint or a vertical vanishing point), that are close to a vanishingpoint, or that align with a vanishing point.

As used herein, the term “ground truth” refers to information of a knownset of pixels reflecting a known set of properties for a digital imageutilized to train a neural network. For example, ground truth vanishinglines can include known vanishing lines identified from a digital image(e.g., lines that point toward a vanishing point). The camera parameterdetermination system can utilize ground truth vanishing lines to train acritical edge detection neural network.

Further, as used herein, the term “training line” refers to linesidentified in a training digital image. In particular, the term“training line” can include a line identified by applying an edgedetection model to a training digital image. Thus, a training line caninclude any line or edge portrayed in a training digital image. Asdiscussed in greater detail below, the camera parameter determinationsystem can determine ground truth vanishing lines from training lines ina training digital image.

Also, as used herein, the term “vanishing edge map” refers to arepresentation of vanishing lines from a digital image. To illustrate, avanishing edge map can comprise an array, vector, database, or image(e.g., black and white or greyscale image) where each entry represents apixel of the digital image and indicates whether the pixel correspondsto a vanishing line. In one or more embodiments, a vanishing edge mapcomprises weights (i.e., a weighted vanishing edge map), where eachweight indicates a confidence value (e.g., a measure of confidence suchas a probability) that a corresponding pixel portrays a portion of avanishing line within the digital image.

Similarly, as used herein, the term “weighted vanishing edge map” refersto a vanishing edge map for a digital image generated based on weightsfor the lines in the image. To illustrate, the camera parameterdetermination system can determine weights for each line in a digitalimage based on various criteria, including line length and/or distancebetween the lines and one or more vanishing points (i.e. angulardistance between the lines in the image and the vanishing points and/orlinear distance between the vanishing points and intersections of linesin the image). Then, the camera parameter determination system maygenerate a vanishing edge map by applying the weights. For example, thecamera parameter determination system can determine lines for inclusionin the weighted vanishing edge map based on the weights (e.g., whetherthe weights satisfy a threshold value).

Also, as used herein, the term “weights” refers to a value utilized toemphasize and/or de-emphasize one or more pixels. For example, a weightcan include a confidence value (e.g., any measure of confidence, such asa probability value). For example, the term “weight” may refer to aconfidence value that a given line is a vanishing line or a confidencevalue that a given pixel is included in a vanishing line. Further, asused herein, the term “training weight” refers to a weight of a lineand/or pixel in a training image. More specifically, a training weightmay be utilized as part of a training image to train a critical edgedetection neural network.

Additionally, as used herein, the term “geometric model” refers to amodel that analyzes lines and/or shapes in a digital image to estimatecamera parameters. For example, the term “geometric model” may refer toa model that groups oriented elements from a digital image and refinesthem iteratively in order to estimate directions or camera parametersfor the image. To illustrate, the term “geometric model” may includemodels that search for converging lines and their intersections todetermine vertical and/or horizontal directions and/or camera parametersin a digital image.

Additional detail will now be provided regarding the camera parameterdetermination system in relation to illustrative figures portrayingexemplary embodiments. In particular, FIG. 1 illustrates a cameraparameter determination environment 100. As illustrated in FIG. 1, thecamera parameter determination environment 100 includes a client device102, including a client application 104, and associated with a user 106.The client device 102 communicates, via a network 108, with the serverdevice(s) 110. The server device(s) 110 can include a digital mediamanagement system 112, which in turn can include the camera parameterdetermination system 114.

Although FIG. 1 illustrates the camera parameter determination system114 implemented via the server device(s) 110, the dynamic representationmanagement system 114 can be implemented via other components. Forexample, the camera parameter determination system 114 can beimplemented in whole, or in part, by the client device 102. Similarly,the camera parameter determination system 114 can be implemented viaboth the client device 102 and the server device(s) 110.

The client device 102 can include various types of computing devices.For example, the client device 102 can be a mobile device (e.g., a smartphone), tablet, laptop computer, desktop computer, or any other type ofcomputing device as further explained below with reference to FIG. 10.Additionally, the client application 104 can include any one of varioustypes of client applications. For example, the client application 104can be an online application (e.g., a web browser), and a user 106 atthe client device 102 can enter a Uniform Resource Locator (URL) orother address directing the web browser to a server device(s) 110.Alternatively, the client application 104 can be a different nativeapplication developed for use on the client device 102.

Additionally, the server device(s) 110 can include one or more computingdevices including those explained below with reference to FIG. 10. Insome embodiments, the server device(s) 110 comprises a content server.The server device(s) 110 can also comprise an application server, acommunication server, a web-hosting server, a social networking server,or a digital content campaign server.

The client device 102, server device(s) 110, and network 108 maycommunicate using any communication platforms and technologies suitablefor transporting data and/or communication signals, including any knowncommunication technologies, devices, media, and protocols supportive ofdata communications, examples of which are described with reference toFIG. 10.

Although not a requirement, the camera parameter determination system114 can be part of a digital media management system 112. The digitalmedia management system 112 gathers, monitors, manages, edits,distributes, and analyzes various media. For example, the digital mediamanagement system 112 can analyze and edit digital images and/or digitalvideo based on user input identified via one or more user interfaces atthe client device 102. In one or more embodiments, the digital mediamanagement system 112 can utilize the camera parameter determinationsystem 114 to determine camera parameters and/or modify digital imagesbased on the camera parameters. For example, the digital mediamanagement system 112 can provide the camera parameter determinationsystem 114 with digital images and the camera parameter determinationsystem 114 can provide the digital media management system 112 withcamera parameters for the provided images. In other embodiments, theserver device(s) 110 can include a system other than the digital mediamanagement system 112, and the camera parameter determination system 114can receive images via alternate means. For example, the serverdevice(s) 110 can receive images via the network 108 from the clientdevice 102 or from another source. As discussed above, the cameraparameter determination system 114 can efficiently, accurately, andflexibly determine camera parameters for a digital image. In particular,FIG. 2 shows an overview of determining camera parameters for digitalimages in accordance with one or more embodiments.

Specifically, as shown in FIG. 2, the camera parameter determinationsystem 114 provides a digital image 202 to a critical edge detectionneural network 204, which generates a vanishing edge map 206. The cameraparameter determination system 114 can then apply a geometric model 208to the vanishing edge map 206. Further, the camera parameterdetermination system 114 can utilize the geometric model 208 todetermine camera parameters 210 for the digital image 202.

As shown in FIG. 2, the camera parameter determination system 114receives a digital image 202 and determines the camera parameters forthat image. The digital image 202 can be any of various file types andcan depict any of various scene types. Further, the camera parameterdetermination system 114 can determine the camera parameters for manydigital images and do so without regard to similarities or differencesbetween the file or scene types of a set of digital images.

As further illustrated in FIG. 2, the camera parameter determinationsystem 114 can utilize a critical edge detection neural network 204 togenerate a vanishing edge map 206. As will be described in greaterdetail with regard to FIGS. 3-4, the camera parameter determinationsystem 114 can train a critical edge detection neural network togenerate a vanishing edge map 206 including the vanishing lines from adigital image. The critical edge detection neural network 204 canidentify lines in the digital image 202 and can determine whether eachof the lines in the digital image is a vanishing line (e.g., a vanishingline corresponding to a vanishing point). As illustrated, the criticaledge detection neural network 204 utilizes the identified vanishinglines from the digital image 202 to generate a vanishing edge map (e.g.,that includes each of the vanishing lines from the digital image 202 andnone of the lines from the digital image that are not vanishing lines).

As shown in FIG. 2, the camera parameter determination system 114utilizes a geometric model 208 to generate camera parameters 210 fromthe vanishing edge map 206. The geometric model 208 detects vanishinglines in the vanishing edge map 206 and use the vanishing lines toestimate vanishing points and camera parameters for the digital image202. Upright can perform this by grouping oriented lines from thevanishing edge map 206 into a number of vanishing points in the sceneand refining them iteratively. Applying the geometric model 208 to thevanishing edge map 206 (instead of the digital image 202) leads to moreaccurate results, because misleading or “false” line segments are notpresent. Accordingly, based on applying the geometric model 208 to thevanishing edge map 206, the camera parameter determination system 114 isable to more accurately and efficiently determine camera parameters 210for the digital image 202.

The geometric model 208 utilizes optimization-based camera calibrationto estimate vanishing lines, vanishing points, and camera parameters fora digital image. The geometric model can also perform uprightadjustments in which the model automatically modifies lines in the imageto straighten the digital image. The adjustments bring slanted linesinto accordance with how human perception would expect to see the image.In other words, the geometric model 208 can remove distortions relativeto human observation. Such adjustment can be helpful in the context ofan image with strong geometric cues, such as images including largemanmade structures.

To determine camera parameters for the image, the geometric model 208performs an edge detection algorithm on the digital image to identifylines in the digital image. The geometric model 208 utilizes an energyfunction, and iteratively optimizes the function to estimate variousmatrices (e.g., a camera intrinsic parameter matrix, an orientationmatrix), which can then be used to estimate the vanishing points,vanishing lines, and camera parameters for an image. However, asdiscussed above, the geometric model 208 may be inaccurate in estimationfor images without strong geometric cues, or with many curved or closeparallel lines.

For example, in one or more embodiments, the camera parameterdetermination system 114 applies the geometric model 208 by utilizingthe approach described by Elya Shechtman, Jue Wang, Hyunjoon Lee, andSeungyong Lee in Camera Calibration and Automatic Adjustment Of Images,U.S. Pat. No. 9,098,885B2, hereby incorporated by reference in itsentirety. Similarly, the camera parameter determination system 114 canapply the geometric model 208 by utilizing the approach described byHyunjoon Lee, Eli Shechtman, Jue Wang, and Seungyong Lee in AutomaticUpright Adjustment of Photographs, Journal of Latex Class Files, Vol. 6,No. 1 (January 2007), hereby incorporated by reference in its entirety.

In one or more embodiments, the camera parameter determination system114 utilizes an alternate approach in response to determining that avanishing edge map is insufficient for the geometric model 208 to yieldaccurate results. For example, in response to determining that avanishing edge map has an insufficient number vanishing lines for use inthe geometric model 208 (e.g., less than a threshold number of vanishinglines that satisfy a confidence value threshold), the camera parameterdetermination system 114 can utilize a direct convolutional neuralnetwork approach. Specifically, in one or more embodiments, the cameraparameter determination system utilizes a convolutional neural network(CNN) based approach to directly determine camera parameters for thedigital image.

As mentioned above, the camera parameter determination system 114 cangenerate ground truth data for training a critical edge detection neuralnetwork. FIGS. 3A-3C illustrate generating ground truth data inaccordance with one or more embodiments. More specifically, FIGS. 3A-3Cillustrate the camera parameter determination system identifying linesin a digital image and determining whether each line is a vanishing lineto generate a set of ground truth vanishing lines for the digital image.In particular, FIG. 3A illustrates the camera parameter determinationsystem 114 generating an unfiltered edge map. Then, FIG. 3B illustratesthe camera parameter determination system 114 mapping the unfilterededge map onto a spherical panorama and identifying major vanishingpoints. Additionally, FIG. 3C shows the camera parameter determinationsystem 114 utilizing major vanishing points and an unfiltered edge mapto generate a ground truth edge map.

As shown in FIG. 3A, the camera parameter determination system 114receives a digital image 302 and generates, based on the digital image302, an unfiltered edge map 304. The unfiltered edge map 304 includeslines detected from the digital image 302, (i.e. both vanishing linesand non-vanishing lines from the digital image 302). To generate theunfiltered edge map 304, the camera parameter determination system 114applies an edge detection algorithm to the digital image 302 to identifyeach of the lines from the digital image.

As shown in FIG. 3B, the camera parameter determination system 114 canmap the unfiltered edge map 304 onto a sphere (i.e., a sphericalpanorama comprising a sphere or a portion of a sphere), generating anedge mapped sphere 306. The edge mapped sphere 306, like the unfilterededge map 304, includes the detected lines from the digital image 302.After mapping the unfiltered edge map 304 onto the sphere, the cameraparameter determination system 114 divides the edge mapped sphere 306into various regions or “bins.” Further, in one or more embodiments, thecamera parameter determination system 114 identifies each intersectionon the edge mapped sphere 306 and, based on these intersections,generate the intersection mapped sphere 308.

As mentioned above, in one or more embodiments, the camera parameterdetermination system 114 may map the training lines from a panoramicdigital image onto a spherical panorama (e.g., a sphere or a sphericalshape). The camera parameter determination system 114 can then “sample”different portions of the panoramic digital image by dividing the imageinto several different, potentially overlapping sub-images. The cameraparameter determination system 114 can prepare these sub-images astraining images and can perform the steps enumerated below with regardto FIGS. 3B-3C and utilize the images as training images when trainingthe critical edge detection neural network.

As shown in FIG. 3B, the camera parameter determination system 114 canperform distance-based voting for vanishing points 310. As discussedabove, in one or more embodiments, the camera parameter determinationsystem 114 can determine a set of “bins” or regions on the spheres 306,308. In one or more embodiments, the camera parameter determinationsystem 114 divides the sphere uniformly or evenly. Further, in one ormore embodiments, the camera parameter determination system 114 candetermine distance-based “votes” for the lines on the edge mapped sphere306 and/or the intersections on the intersection mapped sphere 308. Thecamera parameter determination system 114 can perform a Hough Transformon the detected line segments on the edge mapped sphere 306, or any of avariety of similar transformations.

More specifically, the camera parameter determination system 114 canutilize any of a variety of voting schemes to determine the majorvanishing points for the digital image 302. The camera parameterdetermination system 114 can determine votes from pixels, lines, orintersections based on distance from or orientation with regard to thevarious potential vanishing points. For example, the camera parameterdetermination system 114 can initiate a pairwise voting scheme in whichintersections between lines are considered with respect to each of thelines involved in the intersection. In another embodiment, the cameraparameter determination system 114 can determine angular distancesbetween each of the lines on the edge mapped sphere 306 and each to thepotential vanishing points to determine “votes.”

Finally, based on the voting, the camera parameter determination system114 can determine major vanishing points 312 for the image. In one ormore embodiments, the camera parameter determination system 114identifies a predetermined number of vanishing points (e.g., the topthree bins as the top three vanishing points). In other embodiments, thecamera parameter determination system 114 identifies any number ofvanishing points (e.g., any vanishing points that satisfy a thresholdnumber or percentage of votes). In some embodiments, the cameraparameter determination system 114 identifies vanishing pointscorresponding to mutually orthogonal vanishing directions. For example,horizontal vanishing directions 314, 316, and vertical vanishingdirection 318 are mutually orthogonal.

As shown in FIG. 3C, the camera parameter determination system 114utilizes the major vanishing points 312 and the unfiltered edge map 304to perform act 320 of weighting lines to generate the ground truth edgemap 322. Specifically, the camera parameter determination system 114 canweight each of the lines in the unfiltered edge map 304. The cameraparameter determination system 114 can determine weights (i.e., trainingweights) for each of the lines based on the distances between each ofthe lines and the major vanishing points of the image 302, and/or basedon alignment of the lines with each of the major vanishing points of theimage 302.

These training weights can reflect a probability or confidence valuethat each of the weighted lines are vanishing lines. In one or moreembodiments, the weights are based on the distance from one or more ofthe identified vanishing points and the lines, intersections between thelines, or pixels making up the lines. For example, the weights can bebased on an angular distance between a vanishing point and the line. Inanother example, the weights can be based on linear distances betweenline intersections and the vanishing points. The camera parameterdetermination system 114 can then assign weights based on these measureddistances, and determine weights based on a linear relationship betweenthe distance and the weight itself.

In one or more embodiments, the camera parameter determination system114 can then use the weights for each of the lines to generate theground truth edge map 316. The camera parameter determination system 114can determine which lines to include as vanishing lines in the groundtruth vanishing edge map 322. In one or more embodiments, the weightingcan include a threshold at which a weight is considered a vanishingline. That is, a weight above a predetermined threshold will cause thecamera parameter determination system 114 to include the weighted linein the ground truth vanishing edge map 322, while a weight below thepredetermined threshold will cause the camera parameter determinationsystem 114 to exclude the weighted line from the ground truth vanishingedge map 322.

The camera parameter determination system 114 can utilize furtherclassifications of lines in the digital image. For example, the cameraparameter determination system 114 can utilize multiple (e.g., two ormore) predetermined thresholds, one that determines a high weight withinthe vanishing edge map 316, and one that determines a low weight withinthe vanishing edge map 316. These two thresholds are given by way ofexample, and it will be appreciated that the camera parameterdetermination system 114 could utilize any number of weight thresholdsthat are helpful in the context of the images to be processed. Moreover,as mentioned above, the camera parameter determination system 114 canutilize continuous weights that reflect distance from vanishing points.

Thus, as illustrated in FIG. 3C, the camera parameter determinationsystem 114 generates a set of ground truth vanishing lines as part ofthe ground truth vanishing edge map 322. The camera parameterdetermination system 114 can repeat the acts illustrated in FIGS. 3A-3Cfor a plurality of digital images (e.g., from an existing repository ofdigital images) and generate a plurality of ground truth vanishing edgemaps that reflect the ground truth vanishing lines for each digitalimage.

As mentioned above, the camera parameter determination system 114 canutilize training digital images ground truth vanishing lines to train acritical edge detection neural network. For example, FIG. 4 illustratesthe camera parameter determination system 114 training a critical edgedetection neural network 204 to accurately determine vanishing lines inaccordance with one or more embodiments. In particular, in relation toFIG. 4, the camera parameter determination system 114 utilizes thetraining image(s) 402 and corresponding ground truth vanishing line(s)data 404 to train the critical edge detection neural network 204.

As shown in FIG. 4, the camera parameter determination system 114utilizes training image(s) 402 and the associated ground truth vanishingline(s) data 404 to train the untrained critical edge detection neuralnetwork 406. As discussed above with regard to FIG. 3, the cameraparameter determination system 114 can utilize the training image(s) 402to generate the ground truth vanishing line(s) data 404. In one or moreembodiments, the untrained critical edge detection neural network 406utilizes the training image(s) 402 and the ground truth vanishingline(s) data 404 to learn to accurately identify vanishing lines from adigital image.

Specifically, as shown in FIG. 4, the camera parameter determinationsystem 114 utilizes the critical edge detection neural network 406 togenerate predicted vanishing edge map(s) 408. In particular, thecritical edge detection neural network analyzes the training image(s)402 to generate a plurality of predicted vanishing lines. For example,the critical edge detection neural network can predict, for each pixel,whether the pixel belongs to a vanishing line within the trainingimage(s) 402.

Then, as shown in FIG. 4, the camera parameter determination system 114compares the training vanishing edge map(s) 408 for the trainingimage(s) 402 with the ground truth vanishing line(s) data 404. Morespecifically, the camera parameter determination system 114 compares thetraining vanishing edge map(s) 408 with the ground truth vanishing linesfor the corresponding training image 402 utilizing a loss function 410that generates a calculated loss. In particular, the loss function 410can determine a measure of loss (e.g., a measure of differences) betweenthe ground truth vanishing line(s) data 404 and the vanishing lines inthe training vanishing edge map 408.

Further, as shown in FIG. 4, the camera parameter determination system114 then utilizes the calculated loss to learn to more accuratelyidentify vanishing lines from a digital image. For example, the cameraparameter determination system 114 modifies the neural networkparameters 414 (e.g., internal weights for layers of the neural network)to reduce or minimize the loss. Specifically, the camera parameterdetermination systems utilize back-propagation techniques to modifyinternal parameters of the critical edge detection neural network 406 toreduce the loss resulting from application of the loss function 410.

By repeatedly analyzing training images, generating predicted vanishingedge maps, comparing the predicted vanishing edge maps with ground truthvanishing lines, and modifying neural network parameters, the cameraparameter determination system 114 can train the critical edge detectionneural network 406 to accurately identify vanishing lines from digitalimages. Indeed, in one or more embodiments, the camera parameterdetermination system 114 iteratively trains the critical edge detectionneural network 406 for a threshold amount of time, for a thresholdnumber of iterations, or until a threshold loss is achieved. Asmentioned above, in one or more embodiments, the camera parameterdetermination system 114 utilizes a critical edge detection neuralnetwork that comprises a convolutional neural network architecture. Forexample, FIG. 5 illustrates an exemplary architecture and application ofthe critical edge detection neural network 204 in accordance with one ormore embodiments. As shown in FIG. 5, and as discussed with regard toFIG. 2, the critical edge detection neural network 204 receives thedigital image 202, identifies the vanishing lines in the digital image202, and generates the vanishing edge map 206 including the vanishinglines form the digital image 202.

The critical edge detection neural network 204 can perform pixel-wiseprediction on the digital image 202. In particular, the critical edgedetection neural network can utilize a variant of an hourglass network.Accordingly, in one or more embodiments, the critical edge detectionneural network performs bottom-up processing by subsampling feature mapscorresponding to the digital image 202 and performs top down process byup sampling the feature maps, with combination of higher resolutionfeatures from bottom layers. Instead of using standard residual units(e.g., convolutional blocks) as the basic building block of the criticaledge detection neural network 204, in one or more embodiments, thecritical edge detection neural network 204 includes an inception-likepyramid of features to identify vanishing lines. In this manner, thecamera parameter determination system 114 can capture multi-scale visualpatterns (or semantics) in analyzing the digital image 202.

Generally, the structure of an hourglass neural network includes roughlyequal top-down and bottom-up processing, and many contain two or morehourglass structures so that data is alternately bottom-up and top-downprocessed. This architecture allows the neural network to captureinformation at every scale and combine information across variousresolutions and has seen great success with regard to identification ofobjects in images. The specific structure of the various layers may varydepending on the purpose of the specific neural network but employs aconvolutional-deconvolutional architecture that assists in pixel-wiseprediction. For example, in one or more embodiments, the cameraparameter determination system 114 can utilize an hourglass neuralnetwork as described by Alejandro Newell, Kaiyu Yang, and Jia Deng inEuropean Conference on Computer Vision (2016), Stacked HourglassNetworks for Human Pose Estimation, hereby incorporated by reference inits entirety.

To illustrate, as shown in FIG. 5, the critical edge detection neuralnetwork 204 includes a plurality of pyramid feature units 502. Inparticular, the critical edge detection neural network 204 utilizesvarious pyramid features similar to a pyramid feature network (PFN).Within these pyramid features, the critical edge detection neuralnetwork 204 can include various parallel convolutional layers, similarto those found in a convolutional neural network (CNN) or morespecifically an inception network.

As shown in FIG. 5, the critical edge detection neural network 204includes multiple convolutional layers including various convolutionoperations. For example, FIG. 5 shows an upper convolutional branch 504with a 1×1 convolution operation and a lower convolutional branch 506with a 1×1 convolution operation, a 3×3 convolution operation, andanother 1×1 convolution operation. Together, these convolutionalbranches 504, 506 make up a pyramid feature unit. As illustrated, theupper convolutional branch 504 is in parallel with the lowerconvolutional branch 506. The pyramid feature unit 502 receives inputand performs convolution operations on the input according to each ofthe convolutional branches 504, 506. Then, at the output 508, thecritical edge detection neural network 204 concatenates the results andsends them to the next module. This approach helps to preserve moredetail in vanishing line segments, avoiding blurred or fuzzy heat mapsthat may result from applying more traditional convolutional blocks.

In one or more embodiments, the critical edge detection neural network204 generates the vanishing edge map 206 by determining a confidencevalue for each pixel. The confidence value indicates a measure ofconfidence from the critical edge detection neural network that thepixels correspond to (e.g., are included in or a part of) a vanishingline. In other embodiments, the critical edge detection neural network204 may determine confidence values for each line. The critical edgedetection neural network 204 can utilize these confidence values todetermine which pixels and/or lines from a digital image to include inthe vanishing edge map 206 as vanishing lines. In one or moreembodiments, this determination is based on a predetermined thresholdvalue that the confidence value must comply with for inclusion. Asmentioned above, the camera parameter determination system 114 can then“feed” the vanishing edge map 206 to a geometric model, which will thenonly consider lines included in the vanishing edge map, to determinevarious camera parameters for the digital image.

Further, the critical edge detection neural network 204 can determineweights for the pixels and/or lines in a digital image based on theconfidence values corresponding to those pixels and/or lines. Then,based on these weights, the critical edge detection neural network cangenerate a weighted vanishing edge map that reflects the assignedweights within the edge map itself. That is, in one or more embodiments,the critical edge detection neural network 204 may generate a weightedvanishing edge map that reflects a measure of confidence for each of thevanishing lines included therein. To illustrate, the weighted vanishingedge map can reflect that pixels and/or lines with a higher confidencevalue should be given greater consideration by the geometric model,while pixels and/or lines with a lower confidence value should be givenlesser consideration by the geometric model. Accordingly, the cameraparameter determination system 114 can utilize the geometric model andweighted vanishing edge map to determine camera parameters.

In addition to (or in the alternative to) confidence values, the cameraparameter determination system 114 can also determine weights based online length. In particular, the camera parameter determination system114 can give heavier weights to pixels/lines that correspond to longervanishing lines. Similarly, the camera parameter determination system114 can give reduced weights to pixels/lines that correspond to shortervanishing lines. Thus, the camera parameter determination system 114 canemphasize and/or de-emphasize pixels/lines based on confidence valuesand/or line length.

The vanishing lines in the vanishing edge map 206 can correspond todifferent vanishing points (e.g., vanishing directions). Morespecifically, the vanishing edge map 206 can include vertical vanishinglines (e.g., vanishing lines with intersections corresponding tovertical vanishing directions) and horizontal vanishing lines (e.g.,vanishing lines with intersections corresponding to horizontal vanishingdirections). It will be appreciated that the camera parameterdetermination system 114 can utilize both vertical and horizontalvanishing lines to determine camera parameters for a digital image. Thatis, the critical edge detection neural network 204 can determine bothvertical vanishing lines and horizontal vanishing lines and can generatea vanishing edge map 206 including both types of vanishing lines. Thecamera parameter determination system 114 can utilize this vanishingedge map 206 and a geometric model to determine camera parameters forthe digital image.

As discussed above with regard to FIG. 2, the camera parameterdetermination system 114 can utilize a vanishing edge map and geometricmodel to determine camera parameters for a single digital image. Forexample, FIG. 6 illustrates the camera parameter determination system114 utilizing camera parameters 602 for a digital image 604 to performvarious actions with the digital image 604. For example, the cameraparameter determination system 114 presents a graphical user interface606 for modifying the digital image. Based on user interaction with thegraphical user interface 606, the camera parameter determination system114 can utilize the camera parameters 602 to generate an enhanceddigital image.

For example, as shown in FIG. 6, the camera parameter determinationsystem 114 utilizes the camera parameters 602 to add a chair to thedigital image 604. The camera parameter determination system 114utilizes the camera parameters 602 to properly orient the char withinthe three-dimensional space (e.g., the scene) portrayed in the digitalimage 604. Indeed, as illustrated the camera parameters indicate camerapitch, roll, and yaw which allows the geometric model to properly orientthe chair along properly within the digital image. Though FIG. 6 showsadding an object to the digital image 604, it will be appreciated thatthe camera parameter determination system 114 can perform any of avariety of photo editing functions including deleting objects, modifyinglighting or texture, combining images, or any other editing function.Moreover, the camera parameters can also be utilized to generate athree-dimensional model of the scene portrayed in the digital image.

Further, as shown in FIG. 6, the camera parameter determination system114 provides visual aids for display with the digital image 604 based onthe camera parameters 602 to facilitate photo editing (e.g., by enablingmore precise interaction with the user interface). Specifically, thecamera parameter determination system 114 provides visual linescorresponding to vanishing points within the digital image 604. Thisapproach enables precision in editing, necessitating fewer interactionsbefore achieving a desired result.

Additionally, the camera parameter determination system 114 can utilizethe camera parameters 602 in an image search context. For example, inone or more embodiments, the camera parameter determination system 114utilizes the camera parameters 602 to identify digital images from animage database with similar or the same camera parameters. Specifically,the camera parameter determination system 114 can determine cameraparameters for all of the digital images in the image database. Thecamera parameter determination system 114 can then conduct a searchbased on the determined camera parameters. For instance, the cameraparameter determination system 114 can provide a user interface in whichthe user 106 can specify search parameters based on the one or morecamera parameters (e.g. images with the same pith, roll, and yaw to aninput digital image). The camera parameter determination system 114, canthen identify images that satisfy the search parameters.

Referring now to FIG. 7, additional detail will be provided regardingcapabilities and components of the camera parameter determination system114 in accordance with one or more embodiments. In particular, FIG. 7shows a schematic diagram of an example architecture of the cameraparameter determination system 114 hosted on a computing device 701. Thecamera parameter determination system 114 can represent one or moreembodiments of the camera parameter determination system 114 describedpreviously.

As shown, the camera parameter determination system 114 is located on acomputing device 701 as part of the digital media management system 112,as described above. In general, the computing device 701 may representvarious types of computing devices (e.g. the server device(s) 110 or theclient device 102). For example, in some embodiments, the computingdevice 701 is a non-mobile device, such as a desktop or server. In otherembodiments, the computing device 701 is a mobile device, such as amobile telephone, a smartphone, a PDA, a tablet, a laptop, etc.Additional details with regard to the computing device 701 are discussedbelow with respect to FIG. 10.

As illustrated in FIG. 7, the camera parameter determination system 114includes various components for performing the processes and featuresdescribed herein. For example, the camera parameter determination system114 includes a ground truth vanishing line data engine 702, a criticaledge detection neural network 704, a neural network training engine 706,a camera parameter engine 707, and a data storage 708. Each of thesecomponents is described below in turn.

As illustrated in FIG. 7, the camera parameter determination system 114can include a ground truth vanishing line data engine 702. The groundtruth vanishing line data engine 702 can create, generate and/or provideground truth data to the camera parameter determination system 114. Asdiscussed above with regard to FIGS. 3A-3C, the ground truth vanishingline data engine 702 can generate ground truth vanishing lines for a setof training digital images. More specifically, the ground truthvanishing line data engine 702 can map digital images onto spheres,determine intersections in a digital image, determine vanishing pointsin a digital image, measure various distances in a digital image, anddetermine ground truth vanishing lines for digital images.

Additionally, as shown in FIG. 7, the camera parameter determinationsystem 114 also includes the critical edge detection neural network 704.As discussed above with regard to FIGS. 3-5, the critical edge detectionneural network 704 can determine vanishing lines in a digital image andgenerate a vanishing edge map including the vanishing lines from thedigital image (but not including other lines from the digital image).The critical edge detection neural network 704 can be trained by theneural network training engine 706, as discussed below.

Further, as shown in FIG. 7, the camera parameter determination system114 also includes the neural network training engine 706. The neuralnetwork training engine 706 can train neural networks to perform varioustasks utilizing ground truth data. As discussed in greater detail withregard to FIG. 4, the neural network training engine 706 can minimize aloss function and utilize ground truth data. More specifically, theneural network training engine 706 can train the critical edge detectionneural network 704 to identify vanishing lines and generate vanishingedge maps. The neural network training engine 706 can use ground truthdata from the ground truth vanishing line data engine 702.

Additionally, as shown in FIG. 7, the camera parameter determinationsystem 114 includes the camera parameter engine 707. As discussed ingreater detail above, the camera parameter engine 707 can determinecamera parameters based on a vanishing edge map generated by thecritical edge detection neural network 704. The camera parameter enginecan utilize a geometric model, as described above.

Also, as shown in FIG. 7, the camera parameter determination system 114includes the storage manager 708. The storage manager 708 can storeand/or manage data on behalf of the camera parameter determinationsystem 114. The storage manager 708 can store any data relevant to thecamera parameter determination system 114. For example, the storagemanager 708 can store vanishing line data 710 and camera parameter data712.

FIG. 7 illustrates a schematic diagram of the computing device 701 uponwhich at least a portion of the camera parameter determination system114 can be implemented in accordance with one or more embodiments. Eachof the components 702-712 of the camera parameter determination system114 can include software, hardware, or both. For example, the components702-712 can include one or more instructions stored on acomputer-readable storage medium and executable by processors of one ormore computing devices, such as the client device 102 or serverdevice(s) 110. When executed by the one or more processors, thecomputer-executable instructions of the camera parameter determinationsystem 114 can cause the computing device(s) to perform the methodsdescribed herein. Alternatively, the components 702-712 can includehardware, such as a special-purpose processing device to perform acertain function or group of functions. Alternatively, the components702-712 of the dynamic representation management system 114 can includea combination of computer-executable instructions and hardware.

Furthermore, the components 702-712 of the camera parameterdetermination system 114 may, for example, be implemented as one or moreoperating systems, as one or more stand-alone applications, as one ormore modules of an application, as one or more plug-ins, as one or morelibrary functions or functions that may be called by other applications,and/or as a cloud-computing model. Thus, the components 702-712 may beimplemented as a stand-alone application, such as a desktop or mobileapplication. Furthermore, the components 702-712 may be implemented asone or more web-based applications hosted on a remote server. Thecomponents 702-712 may also be implemented in a suite of mobile deviceapplications or “apps.” To illustrate, the components 702-712 may beimplemented in an application, including but not limited to ADOBE®DIMENSION®, ADOBE® STOCK, ADOBE® PHOTOSHOP®, ADOBE® LIGHTROOM®, ADOBE®PAINTCAN®, ADOBE® LUMETRI™, or ADOBE PREMIERE®.” ADOBE, ADOBE DIMENSION,ADOBE STOCK, PHOTOSHOP, LIGHTROOM, PAINTCAN, LUMETRI, and ADOBEPREMIERE, are either registered trademarks or trademarks of Adobe Inc.in the United States and/or other countries.

FIGS. 1-7, the corresponding text, and the examples provide a number ofdifferent methods, systems, devices, and non-transitorycomputer-readable media of the camera parameter determination system114. In addition to the foregoing, one or more embodiments can also bedescribed in terms of flowcharts comprising acts for accomplishing aparticular result, as shown in FIGS. 8-9. FIGS. 8-9 may be performedwith more or fewer acts. Further, the acts may be performed in differingorders. Additionally, the acts described herein may be repeated orperformed in parallel with one another or parallel with differentinstances of the same or similar acts.

As mentioned, FIGS. 8-9 illustrate a flowchart of a series of acts 800,900 for training and utilizing a critical edge detection neural networkin accordance with one or more embodiments. While FIGS. 8-9 illustrateacts according to one embodiment, alternative embodiments may omit, addto, reorder, and/or modify any of the acts shown in FIGS. 8-9. The actsof FIGS. 8-9 can be performed as part of a method. Alternatively, anon-transitory computer-readable medium can comprise instructions that,when executed by one or more processors, cause a computing device 701 toperform the acts of FIGS. 8-9. In some embodiments, a system can performthe acts of FIGS. 8-9.

As illustrated in FIG. 8, the series of acts 800 includes the act 802 ofidentifying an image captured by a camera with camera parameters. Forexample, the act 802 can involve identifying a digital image capturedvia a camera device with one or more camera parameters. The act 802 canalso involve determining the one or more parameters by applying ageometric model to the first set of vanishing lines and the second setof vanishing lines. Further, the act 802 can involve identifying adigital image utilizing an image search database to identify an imagewith certain characteristics.

Further, as shown in FIG. 8, the series of acts 800 includes the act 804of using a critical edge detection neural network 204 to generate avanishing edge map from the digital image. For example, the act 804 caninvolve utilizing a critical edge detection neural network 204 togenerate a vanishing edge map from the digital image, wherein thevanishing edge map comprises a plurality of vanishing lines from thedigital image corresponding to vanishing points in the digital image,and wherein the critical edge detection neural network 204 is trained togenerate vanishing edge maps from training digital images and groundtruth vanishing lines corresponding to ground truth vanishing points ofthe training digital images. Further, the act 804 may involve using thecritical edge detection neural network 204, wherein the vanishing edgemap comprises the vanishing lines from the digital image. Further, theact 804 may involve using a critical edge detection neural network 204to generate a weighted vanishing edge map that reflects the confidenceand/or probability that each line is a vanishing line. The act 804 canalso include generating the vanishing edge map by generating a first setof vanishing lines with intersections corresponding to a vanishing pointcorresponding to a horizontal vanishing direction and a second set ofvanishing lines having a second intersection corresponding to avanishing point corresponding to vertical vanishing direction.

The vanishing edge map of act 804 may comprise confidence valuescorresponding to pixels of the digital image, the confidence valuescomprising a measure of confidence that the pixels correspond to thevanishing lines. Further, the act 804 may include determining weightsfor the plurality of lines based on the confidence values, generating aweighted vanishing edge map based on the weights for the plurality oflines, and generating the one or more camera parameters based on theweighted vanishing edge map. Further, the critical edge detection neuralnetwork 204 of act 804 may comprise a convolutional neural network.

Additionally, as shown in FIG. 8, the series of acts 800 includes theact 806 of determining, using the vanishing edge map, the cameraparameters for the image. For example, the act 806 can involvedetermining, utilizing the vanishing edge map, the one or more cameraparameters corresponding to the digital image. The act 806 may alsoinvolve determining, using a weighted vanishing edge map, the one ormore camera parameters for the digital image. The act 806 can alsoinclude determining the camera parameters by placing greaterconsideration on pixels and/or line segments given greater weight in theweighted vanishing edge map and lesser consideration on pixels and/orline segments given lesser weight in the weighted vanishing edge map.

Further, the camera parameters of act 806 can comprise at least one offocal length, pitch, roll, or yaw. The act 806 may also involveutilizing a geometric model to determine the one or more cameraparameters for the digital image. Further, the geometric model candetermine the one or more camera parameters for the digital image basedon the vanishing lines included in the vanishing edge map, and notincluding other lines from the digital image. Further, the act 806 mayinvolve utilizing the geometric model to determine the camera parametersfor the image, wherein the geometric model determines camera parametersfor the image based, at least in part, on one or more confidence valuesand/or weights associated with the vanishing lines included in thevanishing edge map.

Moving to FIG. 9, the series of acts 900 includes the act 902 ofdetermining, using training lines in a training image, vanishing pointsfor the training image. For example, the act 902 can involvedetermining, utilizing training lines in the training digital image ofthe plurality of training images, vanishing points for the trainingdigital image. The act 902 can also involve subdividing the trainingdigital image into regions. Further, the act 902 can include determininga predetermined number of major vanishing points based on regionsreceiving the most votes in a voting scheme.

The act 902 can also involve mapping the training lines to a sphericalpanorama, analyzing the training lines to generate a plurality of votesfor a plurality of candidate vanishing regions, and determining thevanishing points from the plurality of candidate vanishing regions basedon the plurality of votes. Additionally the act 902, can involve avoting scheme wherein each pixel in the image votes for one or moreregions of the training digital image as a vanishing point. Further, theact 902 can include a voting scheme wherein each line in the trainingdigital image votes for one or more regions of the image as a vanishingpoint. Further, the act 902 can involve applying a Hough Transformation(or any of a variety of similar transformations) to the training lineson the spherical panorama. Additionally, the act 902 can involvedetermining a vertical vanishing point, a first horizontal direction,and a second horizontal direction for the training digital image.

Also, as shown in FIG. 9, the series of acts 900 includes the act 904 ofgenerating a set of ground truth vanishing lines for the training image.For example, the act 904 can involve utilizing an edge mapped sphere togenerate a set of ground truth vanishing lines for the training digitalimage. Additionally, the act 904 can involve subdividing the edge mappedsphere into a plurality of sub-images and determining vanishing linesfor each of the various sub-images for use as training images for thecritical edge detection neural network.

Also, as shown in FIG. 9, the series of acts 900 includes the act 906 ofdetermining distance between a vanishing point and a training line. Forexample, the act 906 can involve determining a distance between avanishing point of the vanishing points and a training line. The act 906may also involve determining a first angular distance between a firstvanishing line of the vanishing lines and a first vanishing point of thevanishing points. Further, the act 906 can involve determining distancesbetween one or more intersections of one or more line segments and oneor more of the vanishing points.

Additionally, as shown in FIG. 9, the series of acts 900 includes theact 908 of including the training line as a vanishing lie based on thedetermined distance. For example, the act 908 can involve including thetraining line in the set of ground truth vanishing lines based on thedistance between the vanishing point and the training line. The act 908can also involve determining, based on the distance between thevanishing point and the training line, a weight for the training line,and comparing the weight for the training line with a distancethreshold. The act 908 can additionally involve including the trainingline in the set of ground truth vanishing lines based on comparing theweight for the training line with the distance threshold.

Further, as shown in FIG. 9, the series of acts 900 includes the act 910of generating predicted vanishing lines from the training image using acritical edge neural network. For example, the act 910 can involvegenerating predicted vanishing lines from the training digital imageutilizing the critical edge detection neural network 204. The act 910can also include utilizing a pyramid feature unit of a layer of thecritical edge detection neural network, wherein the pyramid feature unitcomprises a convolution operation in parallel with a series of aplurality of convolution operations. Further, the act 910 can involveconcatenating results of the convolution operations from each layer ofthe pyramid feature unit.

Also, as shown in FIG. 9, the series of acts 900 includes the act 912 ofmodifying parameters of the critical edge neural network by comparingthe predicted vanishing lines and the ground truth vanishing lines. Forexample, the act 912 can involve modifying parameters of the criticaledge detection neural network 204 by comparing the predicted vanishinglines and the set of ground truth vanishing lines. Further, act 912 caninvolve modifying parameters of the critical edge detection neuralnetwork 204 utilizing a loss function, wherein the critical edgedetection neural network modifies the parameters based on minimizationof loss determined by the loss function.

In addition (or in the alternative) to the acts describe above, in someembodiments, the series of acts 800, 900 include a step for training acritical detection edge neural network to generate vanishing edge mapsfrom training digital images. The methods and acts described in relationto FIGS. 3A-5 can comprise the corresponding acts for training acritical detection edge neural network to generate vanishing edge mapsfrom training digital images.

In addition (or in the alternative) to the acts describe above, in someembodiments, the series of acts 800, 900 include a step for utilizingthe critical edge detection neural network to generate a vanishing edgemap for the digital image. The methods and acts described in relation toFIGS. 2 and 5 can comprise the corresponding acts for utilizing thecritical edge detection neural network to generate a vanishing edge mapfor the digital image.

Embodiments of the present disclosure may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentdisclosure also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. In particular, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices (e.g., any of the media content access devicesdescribed herein). In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium(e.g., memory), and executes those instructions, thereby performing oneor more processes, including one or more of the processes describedherein.

Computer-readable media can be any available media that can be accessedby a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arenon-transitory computer-readable storage media (devices).Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,embodiments of the disclosure can comprise at least two distinctlydifferent kinds of computer-readable media: non-transitorycomputer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) includes RAM,ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM),Flash memory, phase-change memory (“PCM”), other types of memory, otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmissions media can include a network and/or data linkswhich can be used to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media tonon-transitory computer-readable storage media (devices) (or viceversa). For example, computer-executable instructions or data structuresreceived over a network or data link can be buffered in RAM within anetwork interface module (e.g., a “NIC”), and then eventuallytransferred to computer system RAM and/or to less volatile computerstorage media (devices) at a computer system. Thus, it should beunderstood that non-transitory computer-readable storage media (devices)can be included in computer system components that also (or evenprimarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed by a processor, cause a general-purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. In someembodiments, computer-executable instructions are executed by ageneral-purpose computer to turn the general-purpose computer into aspecial purpose computer implementing elements of the disclosure. Thecomputer-executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, tablets, pagers, routers, switches, and the like. The disclosuremay also be practiced in distributed system environments where local andremote computer systems, which are linked (either by hardwired datalinks, wireless data links, or by a combination of hardwired andwireless data links) through a network, both perform tasks. In adistributed system environment, program modules may be located in bothlocal and remote memory storage devices.

Embodiments of the present disclosure can also be implemented in cloudcomputing environments. As used herein, the term “cloud computing”refers to a model for enabling on-demand network access to a shared poolof configurable computing resources. For example, cloud computing can beemployed in the marketplace to offer ubiquitous and convenient on-demandaccess to the shared pool of configurable computing resources. Theshared pool of configurable computing resources can be rapidlyprovisioned via virtualization and released with low management effortor service provider interaction, and then scaled accordingly.

A cloud-computing model can be composed of various characteristics suchas, for example, on-demand self-service, broad network access, resourcepooling, rapid elasticity, measured service, and so forth. Acloud-computing model can also expose various service models, such as,for example, Software as a Service (“SaaS”), Platform as a Service(“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computingmodel can also be deployed using different deployment models such asprivate cloud, community cloud, public cloud, hybrid cloud, and soforth. In addition, as used herein, the term “cloud-computingenvironment” refers to an environment in which cloud computing isemployed.

FIG. 10 illustrates a block diagram of an example computing device 1000that may be configured to perform one or more of the processes describedabove. One will appreciate that one or more computing devices, such asthe computing device 1000 may represent the computing devices describedabove (e.g., the computing device 701, the server device(s) 110, and theclient device 102). In one or more embodiments, the computing device1000 may be a mobile device (e.g., a mobile telephone, a smartphone, aPDA, a tablet, a laptop, a camera, a tracker, a watch, a wearabledevice, etc.). In some embodiments, the computing device 1000 may be anon-mobile device (e.g., a desktop computer or another type of clientdevice 102). Further, the computing device 1000 may be a server devicethat includes cloud-based processing and storage capabilities.

As shown in FIG. 10, the computing device 1000 can include one or moreprocessor(s) 1002, memory 1004, a storage device 1006, input/outputinterfaces 1008 (or “I/O interfaces 1008”), and a communicationinterface 1010, which may be communicatively coupled by way of acommunication infrastructure (e.g., bus 1012). While the computingdevice 1000 is shown in FIG. 10, the components illustrated in FIG. 10are not intended to be limiting. Additional or alternative componentsmay be used in other embodiments. Furthermore, in certain embodiments,the computing device 1000 includes fewer components than those shown inFIG. 10. Components of the computing device 1000 shown in FIG. 10 willnow be described in additional detail.

In particular embodiments, the processor(s) 1002 includes hardware forexecuting instructions, such as those making up a computer program. Asan example, and not by way of limitation, to execute instructions, theprocessor(s) 1002 may retrieve (or fetch) the instructions from aninternal register, an internal cache, memory 1004, or a storage device1006 and decode and execute them.

The computing device 1000 includes memory 1004, which is coupled to theprocessor(s) 1002. The memory 1004 may be used for storing data,metadata, and programs for execution by the processor(s). The memory1004 may include one or more of volatile and non-volatile memories, suchas Random-Access Memory (“RAM”), Read-Only Memory (“ROM”), a solid-statedisk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of datastorage. The memory 1004 may be internal or distributed memory.

The computing device 1000 includes a storage device 1006 for storingdata or instructions. As an example, and not by way of limitation, thestorage device 1006 can include a non-transitory storage mediumdescribed above. The storage device 1006 may include a hard disk drive(HDD), flash memory, a Universal Serial Bus (USB) drive or a combinationthese or other storage devices.

As shown, the computing device 1000 includes one or more I/O interfaces1008, which are provided to allow a user to provide input to (such asuser strokes), receive output from, and otherwise transfer data to andfrom the computing device 1000. These I/O interfaces 1008 may include amouse, keypad or a keyboard, a touch screen, camera, optical scanner,network interface, modem, other known I/O devices or a combination ofsuch I/O interfaces 1008. The touch screen may be activated with astylus or a finger.

The I/O interfaces 1008 may include one or more devices for presentingoutput to a user, including, but not limited to, a graphics engine, adisplay (e.g., a display screen), one or more output drivers (e.g.,display drivers), one or more audio speakers, and one or more audiodrivers. In certain embodiments, I/O interfaces 1008 are configured toprovide graphical data to a display for presentation to a user. Thegraphical data may be representative of one or more graphical userinterfaces and/or any other graphical content as may serve a particularimplementation.

The computing device 1000 can further include a communication interface1010. The communication interface 1010 can include hardware, software,or both. The communication interface 1010 provides one or moreinterfaces for communication (such as, for example, packet-basedcommunication) between the computing device and one or more othercomputing devices or one or more networks. As an example, and not by wayof limitation, communication interface 1010 may include a networkinterface controller (NIC) or network adapter for communicating with anEthernet or other wire-based network or a wireless NIC (WNIC) orwireless adapter for communicating with a wireless network, such as aWI-FI. The computing device 1000 can further include a bus 1012. The bus1012 can include hardware, software, or both that connects components ofcomputing device 1000 to each other.

In the foregoing specification, the invention has been described withreference to specific example embodiments thereof. Various embodimentsand aspects of the invention(s) are described with reference to detailsdiscussed herein, and the accompanying drawings illustrate the variousembodiments. The description above and drawings are illustrative of theinvention and are not to be construed as limiting the invention.Numerous specific details are described to provide a thoroughunderstanding of various embodiments of the present invention.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. For example, the methods described herein may beperformed with less or more steps/acts or the steps/acts may beperformed in differing orders. Additionally, the steps/acts describedherein may be repeated or performed in parallel to one another or inparallel to different instances of the same or similar steps/acts. Thescope of the invention is, therefore, indicated by the appended claimsrather than by the foregoing description. All changes that come withinthe meaning and range of equivalency of the claims are to be embracedwithin their scope.

What is claimed is:
 1. A non-transitory computer readable storage mediumcomprising instructions that, when executed by at least one processor,cause a computer system to: identify a digital image captured via acamera device with camera parameters; utilize a critical edge detectionneural network to generate a vanishing edge map from the digital image,wherein: the vanishing edge map comprises a plurality of vanishing linesfrom the digital image corresponding to vanishing points in the digitalimage; the vanishing edge map comprises confidence values correspondingto pixels of the digital image, the confidence values comprising ameasure of confidence that the pixels correspond to one or more of theplurality of vanishing lines; and the critical edge detection neuralnetwork is trained to generate vanishing edge maps from training digitalimages and ground truth vanishing lines corresponding to ground truthvanishing points of the training digital images; and determine,utilizing the vanishing edge map, the camera parameters corresponding tothe digital image.
 2. The non-transitory computer readable storagemedium of claim 1, further comprising instructions that, when executedby the at least one processor, cause the computer system to generate thevanishing edge map by generating a first set of vanishing lines withintersections corresponding to a horizontal vanishing direction and asecond set of vanishing lines having a second intersection correspondingto a vertical vanishing direction.
 3. The non-transitory computerreadable storage medium of claim 2, further comprising instructionsthat, when executed by the at least one processor, cause the computersystem to determine the camera parameters by applying a geometric modelto the first set of vanishing lines and the second set of vanishinglines.
 4. The non-transitory computer readable storage medium of claim1, further comprising instructions that, when executed by the at leastone processor, cause the computer system to generate an enhanced digitalimage from the digital image based on the determined camera parameters.5. The non-transitory computer readable storage medium of claim 4,further comprising instructions that, when executed by the at least oneprocessor, cause the computer system to: determine weights for theplurality of vanishing lines based on the confidence values; generate aweighted vanishing edge map based on the weights for the plurality ofvanishing lines; and generate the camera parameters based on theweighted vanishing edge map.
 6. The non-transitory computer readablestorage medium of claim 1, wherein the camera parameters comprise atleast one of focal length, pitch, roll, or yaw.
 7. The non-transitorycomputer readable storage medium of claim 1, wherein the critical edgedetection neural network comprises a convolutional neural network.
 8. Amethod for determining camera parameters for a digital image comprising:identifying a digital image captured via a camera device with cameraparameters; utilizing a critical edge detection neural network togenerate a vanishing edge map from the digital image, wherein: thevanishing edge map comprises a plurality of vanishing lines from thedigital image corresponding to vanishing points in the digital image;the vanishing edge map comprises confidence values corresponding topixels of the digital image, the confidence values comprising a measureof confidence that the pixels correspond to one or more of the pluralityof vanishing lines; and the critical edge detection neural network istrained to generate vanishing edge maps from training digital images andground truth vanishing lines corresponding to ground truth vanishingpoints of the training digital images; and determining, utilizing thevanishing edge map, the camera parameters corresponding to the digitalimage.
 9. The method of claim 8, wherein the camera parameters compriseone or more of focal length, pitch, roll, and yaw.
 10. The method ofclaim 8, wherein the vanishing edge map comprises one or more vanishingline from the digital image.
 11. The method of claim 8, furthercomprising generating an enhanced digital image from the digital imagebased on the determined camera parameters.
 12. A system comprising: oneor more memory devices comprising: a digital image captured via a cameradevice with camera parameters; and a critical edge detection neuralnetwork; and one or more computing devices configured to cause thesystem to: utilize the critical edge detection neural network togenerate a vanishing edge map from the digital image by generating afirst set of vanishing lines with intersections corresponding to ahorizontal vanishing direction and a second set of vanishing lineshaving a second intersection corresponding to a vertical vanishingdirection; and determine, utilizing the vanishing edge map, the cameraparameters corresponding to the digital image by applying a geometricmodel to the first set of vanishing lines and the second set ofvanishing lines.
 13. The system of claim 12, wherein the one or morecomputing devices are further configured to cause the system toreproject the digital image to align the first set of vanishing linesand the second set of vanishing lines to a plurality of vanishingpoints.
 14. The system of claim 12, wherein the vanishing edge mapcomprises confidence values corresponding to pixels of the digitalimage, the confidence values comprising a measure of confidence that thepixels correspond to one or more of the first set of vanishing lines orthe second set of vanishing lines.
 15. The system of claim 14, whereinthe one or more computing devices are further configured to cause thesystem to: determine weights for the first set of vanishing lines basedon the confidence values; generate a weighted vanishing edge map basedon the weights for the first set of vanishing lines; and generate thecamera parameters based on the weighted vanishing edge map.
 16. Thesystem of claim 12, wherein the camera parameters comprise at least oneof focal length, pitch, roll, or yaw.
 17. The system of claim 12,wherein the critical edge detection neural network comprises aconvolutional neural network.
 18. The system of claim 12, wherein theone or more computing devices are further configured to cause the systemto generate an enhanced digital image from the digital image based onthe determined camera parameters.
 19. The system of claim 12, whereinthe one or more computing devices are further configured to cause thesystem to generate, based on the determined camera parameters, one ormore visual aids for display with the digital image during imageediting.
 20. The system of claim 12, wherein the one or more computingdevices are further configured to cause the system to conduct an imagesearch based on the determined camera parameters.